SPECIFICATION 



TO ALL WHOM IT MAY CONCERN: 

BE IT KNOWN THAT I, MASAJIRO IWASAKI, a 
citizen of Japan residing at Kanagawa, Japan have 
invented certain new and useful improvements in 

PRESENTATION OF IMAGES RESEMBLING EACH OTHER 

of which the following is a specification:- 
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BACKGROUND OF THE INVENTION 

1, Field of the Invention 

The present invention generally relates to a 
method of displaying a plurality of images that are 
similar to each other, and relates to a memory medium 
having a program embodied therein for displaying such a 
plurality of images . 

2 . Description of the Related Art 

An increase in CPU speed together with 
increases in memory volumes of main /secondary memory 
devices have made it easier to treat image data. 
Further, widespread use of the Internet and digital 
cameras or the like have made it possible to transfer 
image data to personal computers or the like quite 
easily, resulting in a large volume of image data 
flooding around us in everyday life. Against this 
background, there is a need for a scheme that allows a 
user to search for a desired image from a massive amount 
of image data. 

A conventional scheme for image search 
typically conducts a search by using attribute 
information after manually attaching the attribute 
information to images. This scheme, however, cannot 
cope with a rapid increase of image data. In recent 
years, an effort has been directed to a study on a 
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scheme that automatically extracts image features such 
as color histograms, textures, and shapes from image 
data, and searches for an image resembling a specified 
image by using the extracted image features. This study 
5 has proven to be a success. 

Such image search schemes include a scheme 
that extracts image features from images in advance, and 
extracts image features from an image specified as a 
search key, followed by comparing the image features of 

M 

*Q 10 the specified image with the image features of other 

m 

Ul images to find resembling images. This scheme can show 

Hi a reasonably good performance when the number of image 

\r$ features is small, but suffers a decrease in processing 

q speed as the number of image features increases, 

rij 15 In terms of graphical interface, a typical 

^3 image search system presents search-result images by 

arranging them in a row in a descending order of 
similarity. This kind of display lacks presentation of 
relationships between images that are gauged 
20 quantitatively at an image pattern level. Similarity 

measures of images may not reflect the user's intention 
of his/her search, so that the search results are merely 
one of the factors to be used in the user's decision 
making. There is a need, therefore, for interface that 
25 can assist a user to organize the search results. In 



particular, a resembling image search is, by its 
definition, conducted with ambiguity existing in the 
user's search request, and the user tends to try and see 
as many images as possible. In such a situation, many 
search results need to be presented to the user. 

In consideration of this, an interface that 
presents to the user as many search results as possible 
and asks for the user's judgment is disclosed in "A User 
Interface Visualizing Feature Space for Content -Based 
Image Retrieval," Technical Report of IEICE, IE98-204. 
This interface presents a complete listing of search 
results in such an organized fashion as to allow a user 
to intuitively understand underlining treads of the 
search results. In detail, images obtained as search 
results are subjected to principal component analysis, 
and image features obtained by the analysis are mapped 
into a 2-dimensinal-display space to present the search 
results. Since the principal component analysis is 
applied only to the search-result images rather than to 
all the images of the database, the number of feature 
dimensions is significantly lower, achieving compression 
of information. Further, the search-result images can 
be presented by more accurately reflecting a nature of 
image -set distribution and by more widely spreading the 
search-result images, thereby allowing the user to grasp 
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underlining treads of the search-result images. 

In the field of a relating endeavor, a visual 
interaction scheme is disclosed in "Visual Interaction 
for Exploration in Information Space of Documents," 
5 Journal of JSSST, Vol.13. This scheme introduces an 
idea of dynamic updating of visualized results in a 
visual classification technique where the updating is 
performed in response to a user operation, and the 
visual classification is given by arranging a large 
yg 10 number of documents and keywords based on their mutual 

U| relationships. Presentation of search results in this 

flj visual interaction scheme arranges documents close to 

in 

£ =i each other in a 2 -dimensional or 3-dimensinal space if 

these documents resemble each other according to the 
15 similarity measure of the documents . Such a similarity 
measure is obtained, for example, by using indexes. 

The scheme disclosed in "A User Interface 
Visualizing Feature Space for Content-Based Image 
Retrieval," Technical Report of IEICE, IE98-204 has a 
20 problem in that principal analysis cannot be carried out 
when image features cannot be represented by vector data 
or when image similarity between images cannot be 
represented by a linear function. 

The scheme disclosed in "Visual Interaction 
25 for Exploration in Information Space of Documents," 



Journal of JSSST, Vol ,13 has a drawback in that a large 
amount of computation becomes necessary when the number 
of keywords becomes large, which makes a processing time 
lengthy, thereby making it difficult for a user to grasp 
a nature of the interactively presented visual 
information . 

Accordingly, there is a need for a resembling 
image search scheme that can display search-result 
images at high speed according to similarity measures 
obtained in a feature space even when image features 
cannot be represented by vector data or even when image 
similarity cannot be represented by a linear function. 

SUMMARY OF THE INVENTION 

Accordingly, it is a general object of the 
present invention to provide a resembling image search 
scheme which satisfies the need described above. 

It is another and more specific object of the 
present invention to provide a resembling image search 
scheme that can display search-result images at high 
speed according to similarity measures obtained in a 
feature space even when image features cannot be 
represented by vector data or even when image similarity 
cannot be represented by a linear function. 

In order to achieve the above objects 



according to the present invention, a method of 
displaying images that resemble each other includes the 
steps of extracting image features from images, dividing 
a feature space of the image features into sub- spaces 
having a hierarchical structure, generating a tree 
structure having the sub- spaces as nodes thereof, 
dividing a display space into divided spaces reflecting 
the tree structure, and assigning the images to the 
corresponding divided spaces so as to display the images 
arranged in the display space. 

In the invention described above, image 
features are positioned in the display space in such a 
manner as to reflect relative special relations in the 
feature space. This ensures that images resembling each 
other are displayed close to each other on the display 
screen. Further, the images can be displayed at high 
speed according to similarity measures obtained in a 
feature space even when image features cannot be 
represented by vector data or even when image similarity 
cannot be represented by a linear function. 

According to one aspect of the present 
invention, the method described above is such that the 
step of dividing a feature space includes a step of 
generating clusters having a circle shape by applying a 
clustering method recursively. 
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In the invention described above, use of a 
clustering method reduces the amount of data processing, 
thereby shortening the processing time. 

According to another aspect of the present 
invention, the method as described above is such that 
the clustering method includes the steps of obtaining a 
distance between an image feature and one of the 
clusters closest to the image feature with respect to 
each one of the image features , selecting an image 
feature successively from the image features in an 
ascending order of the distance, obtaining an increase 
in a radius of each cluster such that said each cluster 
contains the selected image feature, and making the 
selected image feature belong to a cluster that has the 
smallest radius increase. 

In the invention described above, more 
accurate clustering can be achieved. 

According to another aspect of the present 
invention, the method as described above is such that 
the step of dividing the display space includes the 
steps of a) dividing the display space in a direction of 
a selected dimension selected for a given node of the 
tree structure into divided spaces as many as there are 
nodes immediately under the given node in the tree 
structure, and b) repeating the step a) by changing the 
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selected dimension as a new node is selected as the 
given node from a next node level. 

In the invention described above, the divided 
display spaces correspond to the respective nodes, and 
5 have the respective images representing the nodes. 

Images that resemble each other are positioned close to 
each other . 

According to another aspect of the present 
invention, the method as described above is such that 
Q 10 the step a) further provides extra spaces between the 

Lf3 divided spaces such that the extra spaces represent gaps 

m 

J3 between the nodes . 

fU 

lf\ In the invention described above, a display 

m 

screen is obtained such that an arrangement of images 

Q 

f» 15 reflects similarity measures more accurately. 

I* According to another aspect of the present 

N 

y invention, the method as described above is such that 

the step a) divides the display space such that the 
divided spaces corresponding to the respective nodes 
20 have sizes proportional to numbers of image features 
belonging to the respective nodes . 

In the invention described above, images tend 
to be evenly distributed over the display space, thereby 
avoiding excessive concentration of images in a given 
2 5 small space and ensuring image display that is easy to 
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see . 

According to another aspect of the present 
invention, the method as described above is such that 
the step a) divides the display space such that the 
divided spaces corresponding to the respective nodes 
have sizes proportional to sizes of the sub-spaces 
corresponding to the respective nodes . 

In the invention described above, a display- 
screen is obtained such that an arrangement of images 
reflects similarity measures more accurately. 

According to another aspect of the present 
invention, the method as described above further 
includes a step of adjusting sizes of the divided spaces 
such that the sizes of the divided spaces corresponding 
to the respective nodes reflect numbers of image 
features belonging to the respective nodes and sizes of 
the sub-spaces corresponding to the respective nodes. 

In the invention described above, a display 
screen is obtained such that an arrangement of images 
reflects similarity measures more accurately. 

Other objects and further features of the 
present invention will be apparent from the following 
detailed description when read in conjunction with the 
accompanying drawings . 



BRIEF DESCRIPTION OF THE DRAWINGS 



Fig.l is a block diagram showing a 
configuration of a resembling image search device; 

Fig. 2 is an illustrative drawing for 
explaining a clustering process; 

Fig .3 is an illustrative drawing showing a 
tree structure of a feature space; 

Fig. 4 is an illustrative drawing showing a 2- 
dimensional presentation of an example of the feature 
space; 

Fig. 5 is an illustrative drawing showing a 3- 
dimensional presentation of an example of the feature 
space; 

Figs . 6A through 6C are illustrative drawings 
for explaining a display-space generation process; 

Fig. 7 is an illustrative drawing showing an 
example of a display screen onto which images are mapped 
according to the present invention; 

Figs . 8A and 8B are illustrative drawings 
showing uneven division of a display space; 

Figs . 9A and 9B are illustrative drawings 
showing another example of uneven division of a display 
space; 

Figs. 1 OA through IOC are illustrative drawings 
showing multi-dimensional division of the display space; 
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and 

Fig. 11 is a block diagram showing a 
configuration of a computer for implementing the present 
invention . 

5 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 

In the following, an embodiment of the present 
invention will be described with reference to the 
accompanying drawings . 
9 10 Fig.l is a block diagram showing a 

Ul configuration of a resembling image search device 1. 

tJl 

*D As shown in Fig.l, the resembling image search 

SU 

Ul device 1 includes an image application 2 and a 

tn 

s resembling image search/display unit 3. The image 

jg 15 application 2 is used for storing and managing image 

ru 

data, and is connected to an image input unit 4 via an 

Q 

g interface unit 5 . The image input unit 4 may be a 

scanner or a digital camera, so that image data input to 
the image input unit 4 is supplied to the image 
20 application 2. The image input unit 4 may be a 

connection to a network, so that image data downloaded 
from a Web site is supplied to the image application 2 . 

The resembling image search/display unit 3 of 
the resembling image search device 1 is connected to a 
2 5 monitor 6 via the interface unit 5. The resembling 
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image search device 1 is implemented by using hardware 
and software. The hardware includes a computer 
comprised of a CPU, a ROM, a RAM, and the like, a hard- 
drive, a CD-ROM drive, etc, as will be described later 
5 with reference to Fig, 11. The software includes system 
software such as an operation system for controlling the 
hardware resources, and includes application programs 
for carrying out various processing by using the 
hardware resources. The application programs are stored 

a 

~J 10 in computer-readable record media such as CD-ROMs, and 

i r? 

?t! are loaded to the computer via the CD-ROM driver or the 

Ml 

like . 

ry 

Uj In the following, the resembling image 

a| 

s search/display unit 3 will be described. The resembling 

r— j 

J2 15 image search/display unit 3 includes a feature 

rtj 

SJ extraction unit 3a, a feature- space tree- structure 

Q 

q extraction unit 3b, and an image display processing unit 

3c. 

The feature extraction unit 3a will be 
20 described first. 

Features suitable for representing image 
characteristics include histogram features, edge 
features, texture features, etc. In the following, a 
description will be given with reference to a case in 
25 which the histogram features are extracted. To extract 
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histogram features, a proper color space (Lab, Luv, HSV, 
etc., for example) is first selected, and is divided 
into a plurality of areas. Then, the number of pixels 
is counted in each area while checking which pixels 
5 belong to which area of the color space, followed by 
normalizing the number of pixels in each area by the 
total number of pixels. This normalized number of 
pixels in each area constitutes the histogram features. 
A distance between two histogram features may be 
10 obtained by calculating a difference of pixel counts 



between an area of one histogram feature and a 

m 

® corresponding area of the other histogram feature and by 

ry 

Ul obtaining a sum of the pixel-count differences for all 

m 

b the areas. Alternatively, a Euclidean distance based on 

S 

45 15 the pixel counts may be obtained. 

fy 

The feature-space tree-structure extraction 

Q 

q unit 3b will be described in the following. 

The feature- space tree- structure extraction 
unit 3b includes a clustering unit 3b- 1 and a tree- 
20 structure generation unit 3b-2. The clustering unit 3b- 
1 divides a feature space into a plurality of sub-spaces 
(clusters) where the feature space is comprised of 
features. Clustering may be carried out by employing a 
Nearest Neighbor method, a K-average algorithm, or any 
25 other method that is known in the art. 
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Fig.2 is an illustrative drawing for 
explaining a clustering process. The clustering process 
will be described below with reference to Fig. 2. 
1. Selecting of Center Features of Clusters 
5 a) Select a feature A in a feature space. 

b) Select a feature farthest away from the 
feature A and designate the selected feature 
as a center feature cl of a cluster. 

c) Select a feature farthest away from the 

Q 10 center feature cl and designate the selected 

U1 feature as a center feature c2 of a second 



cluster. 

ru 

Ul d) Select a feature far away from either one of 

Ul 

~ the selected center features and designate 

Q 

15 the selected feature as a center feature of a 

rij 

*ti new cluster. 

Q 

^ e) Repeat the step d) to obtain center features 

as many as clusters are necessary. 
2 . Sorting of Features 
20 a) Select a feature P in the feature space. 

b) Obtain a distance between the feature P and 
each of the center features and select the 
smallest distance as a cluster distance. 

c) Repeat the steps a) and b) to obtain cluster 
25 distances for features P (P=l, 2, 3, . . . , 



n) . 

d) Sort the cluster distances of the features P 
in an ascending order. 
3. Distribution of Features 

a) Select zero as a radius of each cluster. 

b) Perform the following process on the features 
P successively in an ascending order of the 
cluster distance. Increase a radius of each 
cluster such that the currently processed 
feature P is included within the increased 
radius of each cluster, and select a cluster 
having the smallest radius increase as a 
cluster to which the currently processed 
feature P belongs . 

c) Repeat the step b) for all the features P 
(P=l, 2, 3, . . . , n). 

The procedure described above is followed to 
perform the clustering process. Since a radius increase 
of a cluster is small with respect to features forming a 
dense crowd as shown in Fig. 2, these features tend to be 
grouped into one cluster. 

The tree-structure generation unit 3b-2 of the 
feature-space tree-structure extraction unit 3b will be 
described below. 



The tree-structure generation unit 3b-2 
performs a tree-structure generation process to make 
branches of a tree structure by treating clusters as 
nodes of the tree structure where the clusters are 
obtained by dividing the feature space into sub -spaces 
through the clustering process. The tree- structure 
generation process further clusters each cluster into 
sub-clusters in much the same manner as described above, 
and treats the sub-clusters as nodes of the tree 
structure. This clustering and branching process is 
repeated in a recursive fashion, thereby generating a 
tree-structure of a feature space. 

Fig. 3 is an illustrative drawing showing a 
tree structure of a feature space. 

After the completion of the tree-structure 
generation process, all features belong to respective 
leaf nodes that are at the lowest level in the hierarchy 
Clusters organized into a tree structure in this manner 
each have a representative image that represents the 
cluster. Also, in this tree structure, features 
resembling each other are arranged close to each other. 

Fig. 4 is an illustrative drawing showing a 2- 
dimensional presentation of an example of a feature 
space. Fig. 5 is an illustrative drawing showing a 3- 
dimensional presentation of an example of a feature 
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space . 

As shown in Fig. 4 and Fig. 5, each cluster 
including images similar to each other is comprised of a 
node of an upper-level cluster (CI, C2 , C3) and child 
5 nodes of lower-level clusters (C4 through CIO). Each 
node includes at least one child node. 

In the following, the image display processing 
unit 3c will be described. 

The image display processing unit 3c includes 

Q 10 a display- space generation unit 3c -1 and a display - 

*Q 

IH screen generation unit 3c- 2. The display- space 

U] 

yS generation unit 3c- 1 performs a display- space generation 

fU 

Ul process. 

iri 

a" Figs.6A through 6C are illustrative drawings 

□ 

jg 15 for explaining the display-space generation process. In 

this embodiment, a three-dimensional space is used as a 
r~ space onto which a tree- structure is mapped. 

In the following, the display- space generation 
process will be described with reference to Figs . 6A 
20 through 6C. 

a) Select one dimension of a display space that is 
a space for visually presenting images, and 
divide the display space evenly in the direction 
of the selected dimension so as to have as many 
2 5 divided spaces as there are branches leading to 
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next- level nodes (CI, C2, C3) in the tree 
structure (Fig.6A). 

b) Assign the nodes (CI, C2, C3) to the respective 
divided spaces (Fig.6A). Here, the nodes are 
positioned close to each other when these nodes 
have similar features. 

c) Repeat the steps a) and b) for child nodes (C4 
though CIO) by selecting a different dimension. 



A recursive process as described above maps 
all the nodes of the tree structure onto the display 
space as shown in Fig.6B and Fig.6C. This ends the 
display- space generation process. 

In what follows, the display- screen generation 
unit 3c- 2 of the image display processing unit 3c will 
be described. The display- screen generation unit 3c-2 
performs a display- screen generation process. 

a) Display an image of a leaf node at the center of 
the leaf node where each leaf node mapped onto 
the display space includes an image representing 
the leaf node . 

b) Project images positioned in the 3 -dimensional 
space onto a 2 -dimensional space to generate a 
display screen (e.g., a display window) where 
the images in the 3 -dimensional space have been 
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mapped into the 3 -dimensional space through the 
display- space generation process. When the tree 
structure is directly mapped onto a 2- 
dimensional space by the display- space 
generation process, there is no need to project 
a 3 -dimensional space onto a 2 -dimensional space, 
and the original 2-dimensional space is used as 
a display screen. 

The display screen generated as described 
above is output to the monitor 6 via the interface unit 
5. 

Fig. 7 is an illustrative drawing showing an 
example of a display screen onto which images are mapped 
according to the present invention. 

As shown in Fig. 7, images resembling each 
other are presented as size-reduced images scattered in 
the display screen. The stronger the resemblance 
between images is, the closer the images are, so that 
the images are presented as being related to each other 
in a visually organized manner. To this end, as 
described above, the feature extraction unit 3a, the 
feature-space tree-structure extraction unit 3b, and the 
image display processing unit 3c are implemented as 
application programs, which causes a computer to perform 
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a resembling image display method . 

In Fig. 7, thumbnail images that are displayed 
on the screen may be of any one of the standard file 
formats, and is not limited to images of a particular 
5 file format- Here, the standard file formats refer to 

formats that store image data that are identified as BMP, 
GIF, JPEG, etc. 

In the embodiment described above, the 
display- space generation process of the image display 
Q 10 processing unit 3c divides the display space evenly into 

U1 sub-spaces as many as there are nodes. The present 

U] 

£j invention is not limited to such even division of the 

ru 

01 display space, and the display space may be divided into 

m 

s sub-spaces of different sizes in proportion to cluster 

15 radii of the nodes (clusters), the numbers of features 
included in the nodes, etc. 

Figs.8A and 8B are illustrative drawings 
showing uneven division of a display space. 

As shown in Fig.8A, nodes (clusters) CI, C2, 
20 C3, and C4 have different radii. In such a case, as 

shown in Fig.8B, the display space is unevenly divided 
into sub-spaces having different sizes commensurate with 
the radii of the nodes. This results in generation of a 
display screen that represents similarities more 
2 5 accurately. By the same token, if the numbers of 



features are different from cluster to cluster, divided 
sub- spaces may be given respective sizes proportional to 
the numbers of features. This ensures that images are 
distributed over the display space rather homogeneously, 
thereby preventing images from crowding a space and 
making it difficult to see the images. 

Figs . 9A and 9B are illustrative drawings 
showing another example of uneven division of a display 
space . 

In Fig.9A, clusters CI and C2 are sibling 
nodes, and are separated by a gap SI. As shown in 
Fig.9B, a display space is divided by providing a gap SI 
between the clusters, and no node is assigned to the 
space of the gap SI. Further, the sub-spaces 
corresponding the clusters have sizes commensurate with 
the respective radii Rl and R2 . Such an arrangement can 
generate a display screen that represents image 
similarities more accurately. 

In the embodiment described above, the 
display- space generation process of the image display 
processing unit 3c selects only one dimension of the 
display space, and divides the display space in the 
direction of the selected dimension. The present 
invention is not limited to such an arrangement. 

Figs.lOA through IOC are illustrative drawings 
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showing multi-dimensional division of the display space. 

As shown Fig.lOA, the display space may be 
divided in the three directions of the display space. 
Alternatively, the display space may be divided in two 
directions of the display space as shown in Fig.lOB, or 
may be divided in two directions of the display space 
with the divided sub- spaces having different sizes as 
shown in Fig .IOC. 

Fig. 11 is a block diagram showing a 
configuration of a computer for implementing the present 
invention . 

An input unit 101 is comprised of a keyboard, 
a mouse, a touch panel, etc. , and is used for entering 
information. An output unit 102 outputs various output 
information and information entered by using the input 
unit 101, and is comprised of a display device, a 
printer, or the like. A CPU 103 executes various 
programs. A memory 104 includes RAMs and ROMs. The 
memory 104 stores therein the programs, and, also, 
serves as a temporal storage to store information 
generated during execution of the programs . A memory 
unit 105 includes a hard-drive unit or the like serving 
as a secondary memory storage, and stores therein the 
programs, data, information generated during program 
execution, etc. A memory-media driving unit 106 has a 
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yl 



memory medium MM mounted therein for recording programs 
and data, and reads the programs and data from the 
memory medium to supply them to the memory 104 and/ or 
the memory unit 105. The memory-media driving unit 106 
5 may also be used as a temporal storage for data 

input/output and/or for providing memory space for 
execution of the programs . 

In such a computer configuration, functions 
shown in Fig.l are implemented as program codes, and are 
10 recorded in the memory medium MM such as a CD-ROM. The 



Lfj 

; =i memory medium MM is then mounted in the memory-media 



driving unit 106, and the program codes are loaded into 
the computer to carry out various functions of the 
present invention . 

15 The memory medium MM may be a semiconductor 

medium such as a ROM, an IC memory card, etc. , an 
optical medium such as a DVD-ROM, an MO, an MD, a CD-R, 
etc., or a magnetic medium such as a magnetic tape, a 
flexible disc, etc. 

20 Further, the programs for implementing the 

functions of the present invention may be provided via a 
communication medium rather than via a memory medium. 

Moreover, the present invention has been 
described with reference to presentation of image data. 

25 It should be noted, however, that the present invention 



is applicable to presentation of document data, for 
example, as long as similarity can be measured based on 
features . 

Further, the present invention is not limited 
to these embodiments , but various variations and 
modifications may be made without departing from the 
scope of the present invention. 

The present application is based on Japanese 
priority application No. 11-120946 filed on April 28, 
1999 with the Japanese Patent Office, the entire 
contents of which are hereby incorporated by reference. 



